Sensor performance and weather effects modeling for Intelligent Transportation Systems (ITS) applications

Optical sensors are used for several ITS applications, including lateral control of vehcles, traffic sign recognition, car following, autonomous vehicle navigation and obstacle detection. This paper treats the performance assessment of a sensodimage processor used as part of an on-board countermeasure system to prevent single vehicle roadway departure crashes. Sufficient image contrast between objects of interest and backgrounds is an essential factor influencing overall system performance. Contrast is determined by material properties affecting reflectedradiated intensities, as well as weather and visibility conditions. This paper discusses the modeling of these parameters and characterizes the contrast performance effects due to reduced visibility. The analysis process first involves generation of mherent roadoff-road contrasts, followed by weather effects as a contrast modification. The sensor is modeled as a charge coupled device (CCD), with variable parameters. The results of the sensodweather modeling will be used to predict the performance on an in-vehicle warning system under various levels of adverse weather. Software employed in this effort was previously developed for the US Air Force Wright Laboratory to determine target/ background detection and recognition ranges for different sensor systems operating under various mission scenarios. 1 .O INTRODUCTION This paper presents a method to determine the performance envelope of an imaging sensor and corresponding processing algorithm to avoid single vehicle roadway departure (SVRD) crashes. These crashes are the source of considerable fatalities, injuries and property damage. For example, in 1990, there were over 1.5 million police reported (PR) single vehicle crashes with 16,438 associated fatalities. Single vehicle roadway departures represented approximately 24.2 percent of all PR crashes and 36.9 percent of all crash fatalities in 1990. These statistics were obtained from the 1990 General Estimates System (GES) and Fatal Accident Reporting System (FARS). There are several causal factors associated with single vehicle roadway departure crashes. Some of these factors include: driver inattention ( 1 5.5%), vehicle speed (20%), evasive maneuver ( 1 3.7%), and loss of directional control on road surface (20.2%). These are weighted percentages and include both straight and curved roads. 1 .1 Autonornous driving systems and crash avoidance Addressing these causal factors builds heavily on systems that have been developed for other applications with system functional goals that require lateral and longitudinal control. For example, during the last 10 years, Carnegie Mellon University has been building increasingly competent systems for autonomous driving. The CMU approach has been to develop smart vehicles capable of driving in natural outdoor environments without intervehicle communication or infrastructure modifications. The CMU computer-controlled vehicles now drive themselves without human intervention at speeds up to 55 mph and for distances of over 90 miles on public roads. They are capable of driving both during the day and night on a wide variety of road types and can also sense and avoid obstacles. as well as automatically parallel park. These technologies have been developed as part of ARPA's Unmanned Ground Vehicle (UGV) program. with the goal of reducing the need for human presence in hazardous situations such as battlefield surveillance missions. These technology advances can also be employed to reduce the risk to civilian drivers as part of Advanced Vehicle Control Systems (AVCS). The techniques developed by CMU are suitable both for Automated Highway System (AHS) applications where the vehicle is controlled automatically, and in driver warning systems where the role of the system is to monitor the environment and suggest actions for the human driver.. These techniques include: artificial neural networks for road following, model-based image processing for convoy following, smart obstacle maps based on sonar. ladar and microwave sensor processing and integrated control systems. ALVINN (Autonomous Land Vehicle In a Neural Network) is a perception sjistem which learns to control CMU NAVLAB vehicles by 'I watching" a person drive. ALVIN"s architecture consists of a single hidden layer back propagation network. The input layer of the network is a 30x32 unit two dimensional "retina," which receives input from the vehicle's video camera. Each input unit is fully connected to a layer of five hidden units, which are in turn fully connected to a layer of 30 output units. The output layer is a linear representation of the direction the vehicle should travel in order to keep the vehicle on the road. A \.ideo image from the on-board camera is injected into the input laj2er to drive the vehicle. Acti\'ation is passed forward through the network and a steering command is generated at the output layer. The most active output unit determines the direction in which to steer. To teach the network to steer, ALVINN is shown video images from the on-board camera as a person drives, and noting the steering direction in which the person is current]), steering. The hack propagation algorithm alters the strengths of connections between the netivork nodes so that the network produces the appropriate steering response when presented with a \:ideo image of the road ahead of the vehicle. After about 3 minutes of training while watching a person drive, ALVINN is able to take over and con:inue driving on its own. Because i t is able to learn what image features are important for particular driving situations, ALVIhW has been successfully trained to drive in a wider variety of situations than other autonomous navigation systems which require fixed, predefined features (like the road's center line) for accurate driving. The situations ALVINN networks have been trained to handle include single lane dirt roads, single lane paved bike paths, two lane suburban neighborhood streets, and lined divided highways. In this last domain, ALVINN has successfully driven autonomously at speeds of up to 70 mph, and for distances of over 90 niiles on a highway north of Pittsburgh. Specialized networks are trained for each new road type. The neturorks are trained not only to output the correct direction to steer, but also to estimate its reliability. ALVINN uses these reliability estimates to select the most appropriate network for the current road type. and to switch networks as the road type changes. 1.3 Technical approach for mitigation of SVRD crashes With the preceding sehicle and image processing capabilities as a basis, CMU and Battelle Memorial Institute are currently under contract to the National High\s:ay Traffic Safety Administration (NHTSA) to develop performance specifications for countermeasure systems to prevent roadway departure crashes. Part of this effort involves modeling the combined effects of vehicle dynamics, the sensor, driver, environment and an in-vehicle countermeasure. These components will be modeled as an integrated computer program so that the user may vary the component parameters to determine the overall performance of a candidate counternieasure system. The CMU/Battelle team is assessing four different hardware systems to address the above causal factors. These systems include: a downward looking laser scanner. a fonvard looking camera. sensors to monitor the pavement (e.g., temperature) and a Global Positioning Sensor (GPS) wit11 map matching. This paper treats the forward looking camera and its image processing algoritlu11. The forward looking sensor generates imagery. which can be processed to extract features such as roadway edges, center lines and curve locations. The location of these image features can be combined with a lateral displacement error metric for a vehicle's position \tithin a lane to trigger a Countermeasure signal once a predefined safety threshold is exceeded. The counternieasure would initiate an in-vehicle warning, for example. to avoid a potential crash due to driver inattention or excessive speed while approaching a curve. The optimum point of initiation of a countermeasure signal is impacted by multiple variables, including the combined sensor system and data processing algorithm capabilities to measure and distinguish variations between road and off-road pixels. The procedure to determine sensor/image processing performance under varying conditions involves a three-step process. With the assistance of Battelle personnel, CMU has acquired several sets of roadway imagery, usually under favorable ambient conditions. Battelle then processes selected frames to transform them to specified levels of adverse weather (e.g., rain at 5 millimeters per hour.) CMU personnel utilize these converted frames to ascertain the performance of a particular image processing algorithm for a given environmental state. As the level of adverse weather becomes more intense, i t is expected that a lateral position sensing algorithm, for example, will calculate lateral positions which become increasingly more unreliable. One of the objectives of our program is to provide an analytical representation of this effect. 1.4 Motivation for technical approach Sensor models are envisioned to be of the form: under conditions X, sensor Y has an error rate of 2. For example, the precipitation rate should be related to the lateral position estimation error of a forward looking system. More concretely, such a model might indicate that in a 7mm/hr rain storm, the standard deviation of a lateral position estimation system is 1Ocm. Developing such quantitative performance models for the situation assessment technologies will be a challenge. The reason is that these systems have the potential to be significantly impacted by envirocmental conditions. I t is crucial to determine just how sensitive these systems are to such factors as weather and lighting conditions. Unfortunately, weather and lighting situations are impossible to dictate. We can't simply go out and say "today we are going to measure the performance of system X under conditions of 7nlmlhr rain, and one mile visibility fog at dusk". Such combinations of conditions happen only rarely and for a brief period of time. When a rare combination of circumstances does occur. i t is very difficult to quantify the parameters of the situation (e.g. the rain rate or visibility). Using a sensor system and processing algorithm to follow/track a road is similar to the military tactical problem of detecting a hard target amidst background. In both cases, a minimum tlveshold of intensity or thermal contrast must be measurable between the road (target) and off-road region (background) seen in the sensor system field of view. Whether using the huniaii visual system, or a vehicle-mounted CCD camera feeding data to an automatic road-folloning algorithm, the integrated system comprised of the sensor and image processing must be able to discern road pixels from off-road pixels in the imaged scene. If there is not enough detectable contrast to distinguish